[state sync] migrate to new bucket format by phoenix-o · Pull Request #25925 · MystenLabs/sui

phoenix-o · 2026-03-20T18:29:26Z

Description

migrates state sync to the new bucket format

Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

vercel · 2026-03-20T18:29:30Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
sui-docs	Ready	Preview, Comment	Mar 25, 2026 3:52pm

2 Skipped Deployments

Project	Deployment	Actions	Updated (UTC)
multisig-toolkit	Ignored	Preview	Mar 25, 2026 3:52pm
sui-kiosk	Ignored	Preview	Mar 25, 2026 3:52pm

amnn

@phoenix-o, could you motivate your use of the indexing framework here, as well as some of the changes you need to make to it?

Why does state sync need those changes, and are those needs general to all users of the indexing framework or specific to state sync?
How much of the indexing framework are you using?

It's possible/likely that the best option is to not use the indexing framework, but implement a small/custom interface over object_store instead that caters exactly to the node software's needs.

phoenix-o · 2026-03-23T18:24:33Z

@amnn: State sync currently uses the old ingestion framework. This PR migrates it(last callsite) and utilizes the new checkpoint format.
Only IngestionClient is used to download the checkpoints.
There are not many changes to the framework itself.
A new optional parameter is added remote_store_extra_settings. It's empty by default, so it does not change the behavior of any existing clients.
This allows dynamically passing extra settings to the underlying object store library for more precise configuration.
For instance here StateSync needs to pass down credentials for requester-pays behavior.
I only need this for AWS at the moment, but I updated GCP and Azure as well (can revert if needed)

amnn · 2026-03-23T18:35:16Z

Only IngestionClient is used to download the checkpoints.

If only IngestionClient is used and even within that you only need to be able to fetch checkpoints from an object store, then I'm still leaning towards not using the indexing framework for this, but just implementing a thin wrapper around object_store, and there's a lot hanging off that dependency that you do not need/use. I am also thinking of the fact that we want to move the indexing framework out of the mono-repo at some point and publish it to crates.io.

For instance here StateSync needs to pass down credentials for requester-pays behavior.

Separately, I think this issue has come up before, so maybe it is worth solving in the framework, even if state sync doesn't use it:

@nickvikeras I recall you needed to solve this problem for the analytics indexer, and you were able to do this without updating the framework -- could you explain what you did (and whether you have a path towards generalising it in the framework?)
@rushrs what does the set-up look like for our indexers -- do they use requester pays buckets and if so how do they get configured? I believe we expose configuration via environment variables for this kind of thing, but I'm not sure we use it.

phoenix-o · 2026-03-23T18:49:08Z

@amnn: sounds good, I’ll revert the changes to IngestionClientArgs. Just thought it might be helpful for more precise future configuration of underlying object store clients

amnn

Thanks @phoenix-o, this is now mostly outside of my domain, so will leave the rest of the review to @bmwill or @nickvikeras

amnn · 2026-03-24T11:23:10Z

crates/sui-network/Cargo.toml

 sui-simulator.workspace = true
 sui-protocol-config.workspace = true
-sui-data-ingestion-core.workspace = true
+sui-indexer-alt-framework.workspace = true


Should be able to get rid of this now, right?

I am a little confused at the state of the PR. It looks like Ashok's preference is to move off of the indexing framework's ingestion client and use object_store directly, but we are still using the ingest client in this revision.

Yes, I’m still using the client. Only reverted the changes to the client itself (i.e. I construct object store instance outside)

At that point, why use the ingestion client? You are pulling in all the code for building indexers (and which will one day not even be in the mono-repo) but all you need is the naming scheme for fetching a checkpoint, right?

the point is to minimize code duplication. We’ll need the same functionality in a couple of other places (i.e. sui-tool). Maybe it should live in some other common place. But ok, removing the sui-indexer-alt-framework dependency from this PR

nickvikeras · 2026-03-24T21:28:42Z

Only IngestionClient is used to download the checkpoints.

If only IngestionClient is used and even within that you only need to be able to fetch checkpoints from an object store, then I'm still leaning towards not using the indexing framework for this, but just implementing a thin wrapper around object_store, and there's a lot hanging off that dependency that you do not need/use. I am also thinking of the fact that we want to move the indexing framework out of the mono-repo at some point and publish it to crates.io.

For instance here StateSync needs to pass down credentials for requester-pays behavior.

Separately, I think this issue has come up before, so maybe it is worth solving in the framework, even if state sync doesn't use it:
* @nickvikeras I recall you needed to solve this problem for the analytics indexer, and you were able to do this without updating the framework -- could you explain what you did (and whether you have a path towards generalising it in the framework?)

* @rushrs what does the set-up look like for our indexers -- do they use requester pays buckets and if so how do they get configured? I believe we expose configuration via environment variables for this kind of thing, but I'm not sure we use it.

This was for reading out of the parquet bucket, not the checkpoint bucket, but the code is here -

sui/crates/sui-analytics-indexer/src/indexer.rs

Lines 151 to 169 in 67116b9

    
           // Apply custom headers (e.g., for requester-pays buckets) 
        
           if let Some(headers_map) = custom_headers { 
        
               let mut headers = HeaderMap::new(); 
        
               for (key, value) in headers_map { 
        
                   headers.insert( 
        
                       HeaderName::try_from(key.as_str())?, 
        
                       HeaderValue::from_str(value)?, 
        
                   ); 
        
               } 
        
               client_options = client_options.with_default_headers(headers); 
        
           } 
        
           GoogleCloudStorageBuilder::new() 
        
               .with_client_options(client_options) 
        
               .with_bucket_name(bucket) 
        
               .with_service_account_path(service_account_path.to_string_lossy()) 
        
               .build() 
        
               .map(|s| Arc::new(s) as Arc<dyn object_store::ObjectStore>) 
        
               .context("Failed to create GCS store")

I think we want to do something similar in the indexing framework so we can enable requester pays on our public checkpoint buckets.

nickvikeras · 2026-03-25T16:55:42Z

crates/sui-network/src/state_sync/worker.rs

+    remote_store_options: Vec<(String, String)>,
+) -> Arc<dyn ObjectStore> {
+    let timeout_secs = 5;
+    let client_options = ClientOptions::new()


Do we need to add .with_default_headers() to set the requester pays headers?

Don’t think so. The current sui-data-ingestion-core client doesn’t set it and works with requester-pays buckets. But once accepted, I’ll state-sync the mainnet FN with the new binary and verify that it works

yeah, spun up FN locally. It syncs from the S3 requester-pays bucket

phoenix-o temporarily deployed to sui-typescript-aws-kms-test-env March 20, 2026 18:29 — with GitHub Actions Inactive

phoenix-o force-pushed the state_sync_alt0 branch from 8aca1df to bd8ab7a Compare March 20, 2026 18:31

phoenix-o temporarily deployed to sui-typescript-aws-kms-test-env March 20, 2026 18:31 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs March 20, 2026 18:37 View deployment

phoenix-o force-pushed the state_sync_alt0 branch from bd8ab7a to a61219e Compare March 20, 2026 18:55

phoenix-o temporarily deployed to sui-typescript-aws-kms-test-env March 20, 2026 18:55 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs March 20, 2026 19:01 View deployment

phoenix-o force-pushed the state_sync_alt0 branch from a61219e to b0770b1 Compare March 20, 2026 19:03

phoenix-o temporarily deployed to sui-typescript-aws-kms-test-env March 20, 2026 19:03 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs March 20, 2026 19:05 View deployment

phoenix-o force-pushed the state_sync_alt0 branch from b0770b1 to 85efc3b Compare March 20, 2026 19:49

phoenix-o temporarily deployed to sui-typescript-aws-kms-test-env March 20, 2026 19:49 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs March 20, 2026 19:51 View deployment

phoenix-o marked this pull request as ready for review March 20, 2026 20:11

phoenix-o requested a review from a team as a code owner March 20, 2026 20:11

phoenix-o requested review from amnn, emmazzz, evan-wall-mysten, henryachen, nickvikeras, tpham-mysten and wlmyng March 20, 2026 20:11

phoenix-o temporarily deployed to sui-typescript-aws-kms-test-env March 20, 2026 20:11 — with GitHub Actions Inactive

phoenix-o requested a review from bmwill March 20, 2026 20:11

amnn reviewed Mar 23, 2026

View reviewed changes

phoenix-o force-pushed the state_sync_alt0 branch from 85efc3b to a75e891 Compare March 23, 2026 19:04

phoenix-o temporarily deployed to sui-typescript-aws-kms-test-env March 23, 2026 19:04 — with GitHub Actions Inactive

phoenix-o removed request for emmazzz, evan-wall-mysten, henryachen, tpham-mysten and wlmyng March 23, 2026 19:04

vercel bot deployed to Preview – sui-docs March 23, 2026 19:06 View deployment

amnn reviewed Mar 24, 2026

View reviewed changes

amnn requested review from amnn and removed request for amnn March 24, 2026 11:25

[state sync] migrate to new bucket format

ebd302f

phoenix-o force-pushed the state_sync_alt0 branch from a75e891 to ebd302f Compare March 25, 2026 15:49

phoenix-o temporarily deployed to sui-typescript-aws-kms-test-env March 25, 2026 15:49 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs March 25, 2026 15:52 View deployment

nickvikeras reviewed Mar 25, 2026

View reviewed changes

nickvikeras approved these changes Mar 25, 2026

View reviewed changes

phoenix-o merged commit 2533f2e into main Mar 26, 2026
64 of 65 checks passed

phoenix-o deleted the state_sync_alt0 branch March 26, 2026 13:36

Conversation

phoenix-o commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Release notes

Uh oh!

vercel bot commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amnn left a comment

Choose a reason for hiding this comment

Uh oh!

phoenix-o commented Mar 23, 2026

Uh oh!

amnn commented Mar 23, 2026

Uh oh!

phoenix-o commented Mar 23, 2026

Uh oh!

amnn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nickvikeras commented Mar 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

phoenix-o commented Mar 20, 2026 •

edited

Loading

vercel bot commented Mar 20, 2026 •

edited

Loading